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In this study, a new method was developed to detect student involvement in 
the online learning process. This method is based on convolutional neural 
network (CNN) as a classifier with an emphasis on the preprocessing process 
combined with a new feature in the form of signal magnitude area (SMA) 
thresholding. In this study, the data used as training data is a public dataset 
that emphasizes the decomposition of electroencephalography (EEG) signals 
into individual signal processing. Twenty subjects were taken to be used as 
test data, with each subject watching online learning lectures in the field of 
computer science on three different devices, either with a flat screen, a curved 
screen or a smartphone screen that is smaller than two standard computer 
monitors. Based on the study's results, it is known that the change in screen 
size is inversely proportional to the level of student attention, the smaller the 
screen, the lower the student's attention. For classification results, the model 
equipped with SMA thresholding outperformed the standard classifier by 


thresholding 8.33% with a test set of 20 people. 
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1. INTRODUCTION 

In the face of the COVID-19 pandemic, a series of government regulations were issued to mitigate 
the impact of this virus in the daily lives of the Indonesian people. Minister of Education decree No. 4 of 2020 
states directives that make online learning mandatory for schools and tertiary institutions in Indonesia [1]. This 
considerable change in teaching and learning paradigm raises questions in academia. Although research 
discussing the effectiveness of online learning has been implemented before [2], [3], but on a large scale like 
this it hasn't been done much [4], not only in terms of education but also many preliminary studies conducted 
in the health sector, especially in the field of eye health [5] and ergonomics [6], [7]. Given that online learning, 
which is the primary way of teaching and learning during this pandemic, requires a very large screen time 
compared to normal conditions, the diverse profiles of Indonesian students make the tools used in carrying out 
this process an interesting variable to study. A similar study has been conducted, but the scope is in the work 
environment and was carried out in Korea [8]. In several fatigue studies that are mostly done in the world of 
transportation [9]-[14], on screen time in education has not become a research problem that many people do 
[15]. However, in several previous studies have indeed been implemented on human-computer interaction, the 
application of brain waves has not become a standard parameter to be used as an aspect of assessing the 
conditions that occur, although in several studies the brain wave parameters show a more stable and objective 
value and can be tested by accurate than other biological parameters [16]-[18]. Brain waves are a test parameter 
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that is often used in defense-based research [19], [20], research in the branch of advertising science as well as 
research in the branch of psychology [21]-[25]. This study utilizes what has been done previously by 
researchers at Carnegie Mellon University which has provided a data set that measures the level of engagement 
and student focus on an online learning activity [26]. However, in this study, no good data cleaning was carried 
out, and no statistical tests were carried out on the data before transforming it into a process tensor in the 
classifier. In this study, each research subject will be compared with all available data to make this research 
more objective with the introduction of a thresholding system based on the signal magnitude area (SMA) 
method so that the classifier obtained in the end will be more sensitive to local overfitting than the classifier, 
which does not use a thresholding system which is strengthened by data from previous studies with almost 
similar experiments [27]. In addition to measuring the effectiveness of using SMA, this study will also compare 
the effectiveness of using different screen sizes in online learning. Research related to the effectiveness of 
screen size in learning has been done [28]-[30]. However, in this case the researcher will use brain wave 
parameters and self-assessment to determine the participant's level of focus when using the three different 
screen sizes. 


2. METHOD 
2.1. Research design 

In this study, the author focuses on the effect of screen size on the level of attention and understanding 
of students in distance learning. In this research design, si classification uses a combination of SMA which has 
been modified by combining it with the convolutional neural network (CNN) classifier with multiple input 
variables. Figure 1 shows the location of the sensors that can be observed in brainwave data retrieval. The data 
retrieval results using these sensors will be processed to obtain brainwave signals affecting student focus. In 
general, this experiment is divided into 4 major phases, namely preprocessing data, converting data into 
SMA-scale data, then converting the data into tensors for processing on the CNN network, and the last stage 
is classification using a multivariable CNN classifier using deep learning methods. Preprocessing is the stage 
where the data is cleaned of artifacts and noise using the z-score averaging method to be processed in the next 
phase. Then the next step is to process the data into a signal magnitude scale by taking the median of each data 
and comparing it with a data point. The next step is the data is prepared to be entered into a CNN-based artificial 
neural network after the conversion is complete then the data is forwarded to a CNN-based classifier that has 
been trained previously by public data that focuses on student engagement and focuses attention on the content 
presented on a digital screen [26]. 


Back 


Figure 1. Point of brainwave location [31] 


In the next process, the classifier will classify the participants into the not-engagement class and the 
focused (engaged) class. In the final process, the model will be tested with a parameter matrix which includes 
classical methods such as sensitivity accuracy, which will be combined with an output comparison where the 
predictions of a system will be compared with the original specifications and later, the general system 
performance will be evaluated according to the data in the matrix. Figure 2 shows the steps for classifying a 
person's attention level using this experiment. 
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Figure 2 shows the steps in proving the method offered, the first step in this research is to collect brain 
wave data. This data can be obtained from an electroencephalography (EEG) tool that can record the 
participants’ brain waves and then be normalized using z-score normalization. To remove the noise from the 
brainwave data, a denoising process was carried out using the SMA method to obtain the allowable data value 
for analysis. This research focuses on the preprocessing process, which needs to be addressed in previous 
research and also uses public data tested by previous researchers [32], not using private data whose data validity 
may be doubted. The normalized data is collected in a feature matrix, and the matrix is transformed into data 
that matches the CNN features to be used. The data will later be tested using a modified model using public 
data. This model recognizes two states, namely a state of comfort and a state of discomfort. The class will 
predict one of the two outputs to be used as a reference in evaluating the performance model. Table 1 is an 
example of the results of recording brain waves using EEG. From the results of these recordings, the system 
can record several attributes of the brain waves in the form of attention level, raw data, delta, theta, alpha, beta, 
and gamma signals. 


Z-Score Preprocessing 
ne Normalization Signal Magnitude 
i Area (SMA) 


Data Trans form to 
Tensor 


Classification Dimensional Average Pulling k Kemel 
mputation 


Signal EEG Reduction 


Evaluation Result of Attention 


Model Peformance Prediction Prediction 


Figure 2. System overview 


Table 1. Sample of brainwave data 


No Att Raw Delta Theta Alpha Beta Gamma 
1 77 612 1168234 76559 17671 31863 5930 
2 61 650 1000755 284199 106613 137788 44498 
3 61 555 1000755 284199 106613 137788 44498 
4 69 359 1397841 83854 13089 39550 12335 
5 69 42 1397841 83854 13089 39550 12335 
6 70 -6 220810 42100 10811 6305 1055 
7 81 33 882279 446036 200258 19463 12986 
8 81 213 882279 446036 200258 19463 12986 
9 88 222 538825 90035 248874 10521 5208 
10 88 -43 538825 90035 248874 10521 5208 
12 90 266 486405 532454 148791 21272 4584 
Data format: 
Att : attention level measured by neurosky algorithm Raw : unfiltered EEG brainwave signal 
Delta : signal delta EEG brainwave Theta : signal theta EEG brainwave 
Alpha : signal alpha EEG brainwave Beta : signal beta EEG brainwave 
Gamma : signal gamma EEG brainwave 


In the final stage, the data in this study will be evaluated and tested based on a confusion matrix as a 
test method that will produce system accuracy, focus comparisons when using different devices, and 
comparisons of the CNN method without preprocessing and using SMA preprocessing. To observe what 
signals, need to be taken to observe student focus, Figure 2 is the condition of a person with a signal recording 
range from 0 to 40 Hz. Based on Table 2, it can be seen that the characteristics of the data tested in this study 
where the data will be parsed according to their respective dimensions, some of which symbolize the SMA 
index, which represent the brain waves that have been divided according to the EEG power band standard, 
namely four types of alpha beta delta waves and a scale property attention results from device manufacturers 
and they will be compared with self-assessment scales made by participants on the training dataset. 
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Table 2. Characteristics of EEG brain waves [33] 


m) 


2295 


Brain wave Frequency range (Hz) Occurrence 
Delta 0-4 Condition sleep 
Theta 4-8 Condition going to sleep 
Alpha 8-13 Condition relax, calm, thinking 
Beta 15—40 Condition focus, alert, mental work, anxiety including 


2.2. Preprocessing EEG 

In the initial step, the dataset that will be used as training data will be normalized. Figure 3 is the 
histogram of the dataset that will be used as system training data. Figure 3 is a histogram of the brainwave 
dataset that will be used as training data, where Figure 3(a) shows the results of the attention level data 
recording, Figure 3(b) shows the raw data from the brainwave recording while Figures 3(c)-(f) shows the 
recorded signal to be observed for the training data. 
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Figure 3. Histogram of features dataset (a) attention level, (b) dataset raw data, (c) delta signal, (d) theta 
signal, (e) alpha signal, and (f) beta signal 


The initial and most crucial part of this research is retrieving data from brain waves and ensuring that 
the data taken is ready to be analyzed and included in the classifier. In this process, the normalization process 
will be implemented using the z-score method with the following formula [34]. 

X-X 


Z= (1) 


SDx 


After normalizing the data, there will be some standardized data using the z-score calculation, and 
Figure 4 is the histogram of the normalized data using the z-score calculation. Data normalization is making 
several variables have the same range of values, none of which are too large or too small, to make statistical 
analysis more straightforward. The normalization method used in this study is the z-score, the standard score. 
The essence of this technique is to transform data from values to a broad scale where the mean is zero, and the 
standard deviation is one. The results of normalization are shown in Figure 4, where Figure 4(a) shows the 
histogram of normalized attention level results, Figure 4(b) shows the results of normalization of raw data, and 
Figures 4(c)-(f) shows the results of normalizing the EEG signal. 
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Figure 4. Histogram of data (a) attention level after normalization, (b) raw data after normalization, (c) delta 
signal after normalization, (d) theta signal after normalization, (e) alpha signal after normalization, and 
(f) beta signal after normalization 


After the normalization process is carried out using the z-score method. The results of brain wave 
normalization will be processed using the SMA method to detect outline data so that the data obtained avoids 
overfitting. The following is the formula used for preprocessing using the SMA method [33]. 


SMA = YN" [xa — 2 (2) 


Before entering the machine learning process, the normalized data must first have a threshold so that 
noise does not occur during machine learning processing. This method will later adjust the characteristics of 
the data with machine learning methods and the characteristics of the single band EEG tool used in this study. 
The determination of the threshold or threshold in this study uses the following equation. 


EEGThreshold (x) = AVG (SMAs) + STD (SMAS xy) (3) 

In calculating the threshold using the formula above, the researcher will determine the threshold for 
brain wave data permitted to be processed using the CNN method. The threshold results from this calculation 
are illustrated in Figures 5(a) and 5(b) with the threshold used as a natural evaluator to compare individuals 
and populations. Figure 5(a) shows the recorded brain waves divided using the SMA scale to produce groups 
above and below the threshold, while Figure 5(b) shows the allowable threshold using the SMA scale. The use 
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of the SMA method in this study aims to standardize the signals obtained to improve the accuracy of the 
test results. 
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Figure 5. Example of the frequency of brain wave recordings (a) normal data before the threshold process 
and (b) results threshold using the SMA method 


2.3. EEG classification with CNN 

CNN is a method derived from the development of the multilayer perceptron (MLP) for processing 
two-dimensional data [35]. Figure 6 is a design for applying the CNN method to classify recorded brainwave 
signals as focused or out of focus. CNN is a method that was first developed under the name NeoCognitron by 
a Japanese researcher named Kunihiko Fukushima [36]. The concept developed by Kunihiko Fukushima was 
later developed by a researcher from the USA on behalf of LeCun. LeCun succeeded in developing CNN's 
initial model under the name LeNet in research that discussed number and handwriting recognition. The 
application of the CNN method is getting more and more popular, thanks to the fact that in 2012 Alex 
Krizhevsky won the ImageNet large scale visual recognition challenge 2012 competition using the CNN 
method. This further proves the CNN method as the best object classification method in the image, after 
outperforming other machine learning methods such as SVM [37]. 
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Figure 6. CNN architecture method 


2.4. Testing scenario 

In this study, testing the proposed method will be carried out on 20 participants. Each participant will 
be given approximately 10 minutes to watch learning videos related to the introduction of algorithms and 
programming. In these 10 minutes, the recording duration will only be 5 minutes to reduce data errors at the 
test's beginning and end. This test will be carried out on each participant using three different devices: a curve 
screen, a flat screen and a smartphone. In addition to assessing the recorded brain waves, this study will also 
conduct a self-assessment based on standardized questions with psychological standards. The purpose of this 
self-assessment is that later there will be two sides to the measurement, which will result in the test results 
being accountable. Figure 7 is testing documentation when students carry out the online learning process using 
different types of screens, where Figure 7(a) shows the user using a curve screen, Figure 7(b) using a flat 
screen, and Figure 7(c) using a smartphone screen. 

Figure 7(a) is the process of recording student brain waves during online learning using a curve screen, 
and you can see the student's head using the tool used to record brain waves. Figure 7(b) shows the process of 
recording brain waves using a flat screen and Figure 7(c) shows the process of recording brain waves when 
students use smartphones. The brain recordings from different screens will later be compared to produce which 
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device produces the highest focus when used as a learning medium. Not only comparing, but this study also 
uses the CNN method to classify attention from brain wave recordings and compares it with the accuracy of 
the application of the CNN+SAM method when classifying participants' attention levels. 


(b) 


Figure 7. Brainwave testing and data collection with (a) curve screen, (b) flat screen, and (c) smartphone 
screen 


3. RESULTS AND DISCUSSION 
3.1. Testing with different screen scenario 

In this study, each participant listened to a lecture on the introduction of algorithms and programming. 
Each lecture session in this study was conducted for approximately 10 minutes for each module. Each 
participant was given a different screen for each module to become a random variable in this study. In this test 
with a duration of 10 minutes, brain wave recording will start from the 5" minute to the 10" minute. This was 
done to minimize the imperfection of the data because the participants in the first minute currently adapting to 
the use of the tool. Following the test scenario described, the total number of participants used is 20. Table 3 
is a sample of test data that displays the results of 10 participants’ brain recordings with different screen types 
like curved, flat, and smartphone screens. The average results of each difference in screen size used in this 
study will be presented in Table 4. 

Table 3 shows the results of recording user brain waves while watching learning videos with different 
types of screens: curve, flat, and smartphone, which were observed by recording brain waves and giving a 
questionnaire in the form of a self-assessment to validate the results of brain wave recording. If you look at it 
broadly, it may be obscure that the difference in the level of attention of different devices is obvious. To see 
the effect of the differences in each screen used is presented in Table 4. 

Table 4 shows the average user attention level results when using the curve, flat, and smartphone 
screens. The average value attention level for curved screens is 71, flat screens are 68, and smartphone screens 
are 65. The results of this attention level are also supported by the results of the self-assessment questionnaire, 
which is directly proportional to the average attention level results. In general, the results of this test show that 
the average level of attention when users use curved screen devices is significantly more than flat screens and 
smartphones. This result is because the large and curved screen makes the user comfortable watching the 
lessons given it will indirectly increase the user's level of attention. 


Table 3. Sample test results with different screen types 


Sample Screen type Raw data Alpha wave Beta wave Delta wave etenion Seli assessment 
user level of attention 
001 Curved 612 11804 74577 12935 75 1 
002 Flat 534 22302 28650 12046 75 0 
003 Smartphone 33 25023 18700 32025 60 1 
004 Flat 650 30609 21250 14027 70 1 
005 Flat 507 44000 16500 21250 70 0 
006 Flat -43 57625 17700 23090 76 1 
007 Curved 465 33000 51250 24270 78 1 
008 Curved 272 17890 72000 13210 80 1 
009 Smartphone 80 24020 48730 32025 73 1 
010 Smartphone 12 24021 38650 31020 72 1 


Table 4. Average test results with different screen types 


Number of ; Average Average Average Average Average Self-assessment 
Screen type : ; 
sample user raw data alpha wave beta wave _deltawave _attention level of attention 
20 Curved 75.56 41383.25 24319.39 605787.35 71 18 
20 Flat 73.12 38112.12 23452.78 60782.45 68 16 
20 Smartphone 73.06 34235.67 18456.81 61973.65 65 14 
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3.2. Comparison of CNN classification and CNN+SMA thresholding 

In the experiment, the CNN classifier and the CNN classifier combined with the SMA were developed 
to predict the study's degree of engagement between subjects from brain wave data. The results of predictions 
and experiments that have been carried out will be compared with the results of each student's self-assessment 
to find out whether the predictions made by the classifier are correct, and then statistical calculations are carried 
out, which are used to measure the effectiveness of the model in measuring the level of student involvement. 
Table 5 is the result of testing the application of the SMA method as preprocessing to classify the level of 
student engagement when doing online learning. 

Table 4 shows the SMA method combined with the CNN method to classify whether the user is 
focused when given video learning material. Based on tests involving 20 users, it was found that the prediction 
results given by the system by reading brain waves using the CNN method were smaller than the combination 
of the CNN and SMA methods. The prediction results from a system that combines the CNN and SMA methods 
have a smaller prediction error than the CNN method, as evidenced by the answers from the user's direct self- 
assessment. Based on system testing, it was found that using the CNN method combined with SMA at the 
preprocessing stage has a higher accuracy value than relying solely on the CNN method, where combining 
these methods can provide an increase of 8.33%. 


Table 5. Comparison of CNN and CNN + SMA method 
Prediction of attention Prediction attention level © CNN accuracy CNN + SMA 


No Screen type level with CNN with CNN + SMA (%) accuracy (%) 
True False True False 
1 Curved 17 3 18 2 85 90 
Flat 14 6 17 3 70 85 
3 Smartphone 15 5 16 4 75 80 
Average Accuracy 76.67 85 


4. CONCLUSION 

Based on the study results, adding a SMA as a thresholding method at the preprocessing stage in the 
development of an engagement detection system in online learning increased the classifier's performance by 
8.33%. The results of experiments conducted in this study indicate a correlation between the addition of the 
SMA index with the addition of the level of attention with a proprietary algorithm produced by Neurosky as a 
device manufacturer. These results indicate that the potential of the SMA method to be developed as a 
parameter to calculate brain wave relaxation is work that needs to be explored in further research. In addition, 
using a curve screen produces a higher attention level value than using flat screens and smartphones in the 
implementation of distance learning. Education providers can consider it one of the concerns to improve the 
quality of student learning. 
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